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PREFACE 



High priority is now being given to the improvement of children's learning 
in the cognitive domain, particularly in connection with productive thinking 
skills. The federal government, various profit and non-profit agencies, and 
many individuals are attempting to extend knowledge about this aspect of 
human learning and to improve related educational practices through programs 
of research and development. Behavioral scientists and school people are 
joining together in an attempt to learn more about the nature of productive 
thinking skills and the conditions associated with more efficient learning of 
them. This report of a large-scale controlled experiment involving personnel 
from three different organizations shows how specialized knowledge and re- 
sources can be brought to bear upon extending knowledge and improving edu- 
cational practice through research and development activities. Personnel 
participating in the study were from the Wisconsin Research and Development 
Center for Cognitive Learning, the Creative Thinking Project of the Institute 
of Personality Assessment and Research at the University of California, 

Berkeley, and the Racine (Wisconsin) Unified School District, No. 1. 

The instructional materials used in the experiment — The Productive Thinking 
Program. Series One: General Problem Solving— were developed by Professor 

Martin Covington, Professor Richard Crutchfield, and Mrs. Lillian Davies of 
the Creative Thinking Project. Dr. Robert Olton, of the same project, was 
chiefly responsible for the development and subsequent scoring of the large 
battery of productive thinking tests used in the experiment and wrote the 
first draft of the major portion of this report. Both the instructional and test 
materials are copyrighted; the instructional materials are now available for 
regular school use. Professor Covington and Dr. Olton, in addition to their 
role in the development of the new materials, worked directly with the staff 
of the Wisconsin R & D Center in planning, executing, and reporting the ex- 
periment. 

Dr. William Goodwin of the Wisconsin R & D Center took primary responsi- 
bility in working out the design of the experiment with Dr. Olton. Dr. Goodwin 
coordinated the various components of the experiment, making sure that the 
pretesting, the instruction, and the posttesting were carried out according to 
schedule during the second semester of the 1965—1966 school year. Dr. James 
Wardrop of the Wisconsin R & D Center was responsible for the computer 
analysis of the data during the 1966—1967 school year, and for writing the 
first draft of the results during the spring and summer of 1967. Professor 
Herbert J. Klausmeier of the Wisconsin R & D Center initially consulted with 
Professor Crutchfield regarding the experiment in the summer of 1965 and 
subsequently developed and directed the strategy for bringing the resources 
of the three organizations to bear most effectively upon execution of the experi- 
ment. He consulted with relevant personnel on all matters regarding the design 
and conduct of the experiment, per se. He and Professor Crutchfield defined 
the more precise roles of the staff and the contribution of their two respective 
organizations; he and Mr. Harris Russell of the Racine schools similarly 
defined the roles and contributions of their two respective organizations. 



Mr. Harris Russell, Director of Instruction, Racine Unified School District 
No. 1, made the necessary arrangements for his central staff, building princi- 
pals, and teachers first to decide whether to participate in the experiment and, 
subsequently, to carry it out. Miss Teckla Ronda of the Racine schools esti- 
mated the extent to which each participating classroom represented a favorable 
creative environment, made sure that material^ were received by teachers and 
used according to the plan outlined, and secured relevant information regarding 
each child, including IQ score, educational achievement test scores, socio- 
economic status, and sex. The. elementary school principals and fifth-grade 
teachers gave generously of their time, and also offered many useful ideas 
concerning the experiment. 

Through this union of personnel and resources from the three organizations, 
answers to the following questions were secured. 

1. To what extent can specified productive thinking abilities be taught to elementary 
school children through the use of programed instructional material designed to 
teach thinking skills independently of the content of any specific subject field ? 

2. Are some productive thinking skills more amenable to instruction than others ? 

3. Can productive thinking skills be learned by children of all IQ levels ? 

4. Are there consistent sex differences in productive thinking skills ? 

5. How are productive thinking skills affected by a classroom environment 
judged to facilitate creative expression by the students ? 

The reader will find this report interesting from two viewpoints. First, it 
is a clear account of the development of productive thinking skills in school 
children; answers to the preceding questions were obtained and are more defini- 
tive than in most experiments. Second, the execution of the experiment, includ- 
ing the placement of classes into experimental and control groups, the type of 
information gathered, the execution of the treatments, and the analysis of the 
data, shows that useful information can be obtained through carefully conducted 
research in school settings. 
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ABSTRACT 



The major purpose of this study was to investigate the extent to which 
increments in the thinking and problem-solving performance of fifth-grade 
students could be produced by the use of self-instructional programed 
lessons (The Productive Thinking Program. Series One: General Problem 
Solving ) which were designed to teach skills and strategies of creative think- 
ing independent of any specific subject field. The study also investigated 
the relationship between productive thinking abilities and certain character- 
istics of the learner (IQ and sex), and the relationship between level of pro- 
ductive thinking performance and the extent to which overall classroom 
"environment'^was judged to facilitate creative thinking. 

Results from 44 fifth-grade classes showed that The Productive Thinking 
Program produced statistically significant increments in thinking and prob- 
lem-solving performance on a wide variety of productive thinking measures. 
These instructional benefits occurred for virtually all types of students 
(regardless of sex or general level of IQ), and were especially marked for 
students in classrooms having environments which were judged to provide 
relatively little support and encouragement for the development of productive 
thinking. These effects were obtained when the materials were used as an 
entirely self-contained, self-instructional program; considerably greater 
educational benefits could be expected under conditions where the materials 
are reinforced by active teacher participation. 

Quite apart from the effects of the instructional materials, performance 
on the productive thinking measures used in this study was significantly 
related to sex (girls generally scoring higher than boys) and showed a 
strong and positive relation to IQ. 
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RATIONALE OF THE STUDY 



BACKGROUND 

During the past five years, the Creative 
Thinking Project has been developing a set of 
special instructional materials designed to de- 
velop and facilitate creative thinking in students 
of the upper grades of elementary school. These 
materials (described in Section II) are a series 
of self- administering programed lessons which 
require a total of about 16 classroom hours for 
completion. The materials have undergone ex- 
tensive pilot development and revision and 
were used in two recent experimental studies 
involving several schools ir. the San Francisco 
Bay area. In both of these studies, the ma- 
terials produced a substantial increase in the 
students' creative thinking performance (Cov- 
ington and Crutchfield, 1965; Crutchfield, 

1966) and were very favorably received by both 
students and teachers. After the second San 
Francisco Bay area study, the materials under- 
went a complete revision, and the desirability 
of a large-scale experimental study with the 
revised materials (titled The Productive Think- 
ing Program. Series One: General Problem 
Solving) became apparent. 

Researchers at the University of Wisconsin 
have also studied creativity in school children, 
particularly in gifted children (Klausmeier, 
Harris, and Ethnathios, 1962; Klausmeier and 
Wiersma, 1964), and have organized programs 
for the gifted which included education for 
creativity (Klausmeier, 1963; Klausmeier and 
Teel, 1964). In addition, the focus of the 
Wisconsin Research and Development Center 
for Cognitive Learning is on improving educa- 
tion through a better understanding of cognitive 
learning, including both creativity and problem- 
solving processes. 

Thus, the directors of the Creative Thinking 
Project and the Wisconsin R & D Center agreed 
that an outstanding research opportunity could 
be realized by pooling some of the specialized 
resources and capabilities of each group in a 
large-scale experimental study. The officials 
of the Racine (Wisconsin) Unified School Dis- 



trict endorsed the study enthusiastically since 
it offered the possibility of extending the pro- 
gram of creativity to a large segment of the 
entire school population. 



AIMS OF THE STUDY 

The major purpose of this study was to in- 
vestigate the extent to which increments in the 
thinking and problem-solving performance of 
fifth-grade students could be produced by the 
use of programed lessons (Series One of The 
Productive Thinking Program ) which were de- 
signed to teach skills and strategies of creative 
thinking independent of any specific subject 
field. 

Within this framework, several specific aims 
were incorporated: 

(1) To investigate further the extent to which 
productive thinking (i. e. , creative thinking 
and problem solving) abilities can be taught to 
elementary school children independent of any 
specific subject field. Currently, there is 
strong emphasis in the schools upon specific 
subject matter areas, with little attention being 
given to teaching general skills which transcend 
specific curricula. If such cognitive skills can 
be taught successfully, they may be applicable 
in a variety of subject areas and should enhance 
performance in these areas. 

(2) To discover whether some of these skills 
or behaviors are more readily taught than others. 
For example, one aspect of problem solving is 
the asking of appropriate questions. Can stu- 
dents be taught this behavior more or less suc- 
cessfully than they can some other skill, such 
as generating a set of possible solutions to a 
problem ? 

(3) To investigate the relationship between 
certain learner characteristics (intelligence and 
sex) and performance on a broad diversity of 
productive thinking measures. It is generally 
reported that girls perform slightly bettor than 
boys on most measures of academic achieve- 
ment. Is this also true of measures of productive 
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thinking ? It has also been reported that intel- 
ligence and "creativity" are relatively independ- 
ent. Does this independence persist throughout 
several different aspects of productive thinking 
and on several different types of problems ? In 
addition, is there a relation between intelligence 
or sex and the ability to learn productive think- 
ing skills ? 

(4) To investigate extent to which overall 
classroom "environment" facilitates productive 
thinking in students, and the extent to which 
students in different types of classroom envi- 
ronments profit from instruction in productive 
thinking. To what extent does merely providing 
an encouraging environment promote creative 
thinking ? Are instructional materials more (or 
less) effective in such environments ? 

(5) To probe some of the instructional limits 
of Series One of The Productive Thinking Pro- 



gram by using these materials as an entirely 
self-contained program of instruction, with all 
forms of teacher participation purposely held 
to a minimum— in much the same way that a 
teacher might assign a program of independent 
reading which a particular student or group of 
students would be expected to do on their own, 
with a minimum of supervision. Moreover, the 
entire program was used in a compressed period 
of time at the rate of virtually one lesson per 
day. Thus, this study investigated the effec- 
tiveness of these materials when they were 
used under the most demanding conditions 
likely to be encountered in a school setting. 

(In addition to being valuable in a practical 
sense, the results would also become an im- 
portant component of a systematic research ef- 
fort to explore the effectiveness of materials 
such as these under a variety of instructional 
conditions. ) 
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II 

METHODS 



SUBJECTS AND CONDITIONS 

Subjects for this experiment were the stu- 
dents in 44 of the 47 fifth-grade classrooms in 
the Racine, Wisconsin, Unified School District 
No. 1. All students in these 44 classes par- 
ticipated in the experiment, but analysis is 
based upon a sample of eight males and eight 
females selected from each class. Selection 
was accomplished by rank-ordering all male 
students from a given classroom on thq basis 
of IQ, and then randomly selecting one student 
from each eighth of the distribution. Eight fe- 
males were then selected from the same class 
in the same way. This procedure was continued 
for all 44 classes. The entire analysis involved 
a total of 704 students. Selection was made 
after the experiment had been completed but 
before any scoring of tests was begun. 

This large sample reflected, in proper pro- 
portion, the wide variety of abilities and back- 
grounds characteristic of students in an urban 
school system (a city of about 100, 000 popu- 
lation) and hence provided a thoroughly repre- 
sentative base for analyzing both overall and 
internal effects. The size of the sample also 
permitted a very stable estimate of error and 
made possible the use of classroom means 
(rather than individual student scores) as the 
unit of analysis, thus reducing sporadic varia- 
tion and also meeting the statistical require- 
ments of random assignment. 

Several weeks before the experiment began, 
the teaching supervisor of the district observed 
each class while the teacher and students were 
having an extended discussion about an inter- 
esting event in social studies, an event which 
provided an opportunity for free, creative inter- 
change between teacher and students. All 
classes discussed the same event, and on the 
basis of her observations, the supervisor re- 
corded the extent to which the overall class- 
room atmosphere (including the teacher) seemed 
to provide an environment that facilitated and 
fostered creative thinking on the part of the 
students. The supervisor reported that she felt 



these ratings were representative of the normal 
state of affairs that existed in the various 
classrooms and were not an artifact of the par- 
ticular discussion situation in which they were 
obtained. The final, single, Environment rat- 
ing assigned to each class was a composite 
score derived from the several objective ratings 
the supervisor made. 

Classrooms were rank-ordered according to 
the composite environment rating. The twenty- 
two classrooms with the highest ratings were 
designated as Facilitative environment class- 
rooms; likewise, those 22 classrooms whose 
composite rating was below the median weie 
classified as Nonfacilitative environment groups. 

Within each of these Environment blocks, 
half the classrooms were randomly assigned to 
the Treatment group (which would use The Prg r 
ductive Thinking Program ) and the remaining half 
constituted the Control group. Thus, the class- 
room population was subdivided as follows: 

Number 
of Classes 

Treatment H 

Facilitative Control 1 !• 

Treatment H 

Nonfacilitative Control If 



INSTRUCTIONAL MATERIALS 

The Productive Think ing Program (Covington, 
Crutchfield, and Davies, 1966) is a set of in- 
structional materials designed to develop and 
strengthen the productive thinking skills of 
students in the upper grades of elementary 
school. The Program undertakes to develop 
the student's ability to use important skills 
anH strategies of thinking and problem solving, 
seeks to enhance his understanding, of the pro- 
cesses of his own thinking, and aims to pro- 
mote favorable attitudes an d motivations toward 
activities which involve the use of the mind. 
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Series One of this Program, which deals with 
General Problem Solving, served as the instruc- 
tional materials in this study. 

The skills taught by Series One of The Pro- 
ductive Thinking Program are those which are 
essential for the kind of complex and original 
thinking in which an individual strives to pro- 
duce new ideas, fresh insights and under- 
standings, and his own solutions to problems. 
Hence the Program teaches the student how to 
generate many ideas, especially clever and 
unusual ones, how to look at a problem in a 
different and more fruitful way if he gets 
"stuck, " and how to use the implications of 
crucial facts or events to gain new perspective 
on a problem which seems to resist solution. 
Moreover, although productive thinking is not 
entirely a matter of logical reasoning, it is 
obvious that reasoning and planfulness play 
an important role in this kind of thinking. 
Consequently, the Program also shows the 
student how to proceed in an organized and 
systematic way when attacking a problem, 
how to clarify the essentials of a problem, and 
how to pay attention to relevant facts and con- 
ditions of the problem in evaluating one's 
ideas. 

In addition to teaching certain essential 
thinking skills, the Program seeks to promote 
those motivations and attitudes that are re- 
quired for effective performance in productive 
thinking tasks. It attempts to increase the 
student' s interest in and liking for activities 
which involve the use of the mind, to build up 
his readiness to work on thinking tasks in a 
persistent and concentrated way, and to 
strengthen his self-confidence in his ability 
to think. 

Description of the Materials 

Series One of The Productive Thinking Pro- 
gram consists of 16 programed lessons, each 
of which is an individual booklet about 40 
pages long. The lessons have a cartoon-text 
format which serves to heighten interest and 
at the same time permits clarity and emphasis 
in the presentation of the material. (Four 
sample pages from one of the lessons are pre- 
sented in the Appendix. ) The lessons are 
written primarily for fifth- and sixth-grade 
students and can be used by students with a 
wide range of intelligence within these grades. 
Each lesson is self-administering, permitting 
the student to work at his own pace; virtually 
all students complete a lesson within 50 
minutes. 



Each lesson is centered about a complex 
and engaging problem, presented in story form, 
which the student attempts to solve. The 
problem features a mysterious occurrence, and 
as the student works on this mystery, he is 
called upon to generate ideas, to look for and 
evaluate the meaning of puzzling facts or dis- 
crepancies in information, to ask questions, 
and to practice other productive thinking skills. 

On certain pages, the student writes out 
his questions and ideas. Feedback to his re- 
sponses is then provided in following pages, 
where he finds a range of illustrative ideas 
appropriate to the problem at this point. In 
this way, he is led to understand what consti- 
tute relevant, fruitful, and original ideas, and 
his readiness to generate such ideas is en- 
hanced. Each lesson is so designed that the 
student works gradually toward a solution to 
the problem and eventually is brought to dis- 
cover it by himself. Thus, he experiences 
the pleasurable excitement which accompanies 
successful problem solution and acquires a 
growing sense of competence in coping with 
difficult and complex problems. 

In addition to giving the student guided 
practice in using productive thinking skills, 
the lessons also give some direct, didactic 
instruction about thinking and about each of 
the skills which the student is being taught. 
This direct instruction is incorporated into 
each lesson as the central problem develops 
and as the student works toward solution. In 
addition, the major points taught in each les- 
son are summarized for the student at the end 
of that lesson. 

The human element in this series of lessons 
is provided by a continuous story-line built 
around the Cannon family of Elmtown, USA. 

Lila and Jim Cannon are in the fifth grade and 
sixth grade, respectively, and their Uncle 
John is a high school science teacher who 
occasionally dops some high-level detective 
work for the Elmtown police. Uncle John 
teaches Jim and Lila how to become better 
thinkers as they work on various "cases" to- 
gether. 

Jim and Lila are intended as counterparts 
of the students who work on these lessons. 

They are typical children with whom the stu- 
dent can identify— likeable and human. Their 
thinking provides examples of how to attack 
problems, and— in some cases — how not to 
attack them. Jim and Lila are intended to 
serve as stimulating "companions" for the 
reader as he pursues these intellectual adven- 
tures, and in many cases the student works on 
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the problems in concert with Jim and Lila— first 
the student generating his own ideas and ques- 
tions, then Jim and Lila responding with theirs. 

Administration of the Materials 

The 16-lesson series requires a total of 16 
classroom hours of instruction time. In the 
present study, the lessons were administered 
one per day for 4 days of each week; the re- 
maining school day in each week was used for 
makeups. Generally, each student worked in- 
dividually at his own pace and had an assign- 
ment to which he turned when he had completed 
the day's lesson. 

It was the intention of this study that the 
teacher's role in the use of these materials be 
held to a minimum so that any posttest superi- 
ority of students who had used the materials 
(as opposed to the controls, who had not) would 
result from the effectiveness of the materials 
themselves, rather than from whatever effects 
might occur when they served as raw material 
fd>r imaginative treatment by capable teachers. 
Thus, the teachers were instructed only to 
distribute the lessons and answer procedural 
questions. They were not to conduct class 
discussions based on the materials or otherwise 
participate in the way they were used. One ex- 
ception to this policy was permitted in classes 
with a number of poor readers. In these classes, 
teachers often read portions of the lessons 
aloud with small groups, or otherwise assisted 
students with substantial reading problems. 



MEASURING INSTRUMENTS AND SCORING 
PROCEDURES 

Most of the measuring instruments used in 
this study were developed by the Creative 
Thinking Project at Berkeley. 1 Since they differ 
from traditional educational tests in both con- 
tent and structure, it is helpful to present a 
brief description of their underlying rationale 
before describing the instruments themselves. 

A more complete discussion of this rationale 
may be found in Covington (1968). 

Briefly, the instruments are based on the 
proposition that creative thinking can be con- 



1 Four tests in the battery (Circles, Squares, 
Truck Improvement, and Dog Improvement) were 
developed by Dr. E. Paul Torrance while at the 
University of Minnesota. His permission to 
use those tests as part of this study is grate- 
fully acknowledged. 



ceptualized as an intricate and highly complex 
problem-solving process; one which requires 
the individual to use and to coordinate a number 
of different cognitive skills as he seeks to reach 
his goal, whether this goal is to create an in- 
sightful explanation, to design an effective 
course of action, or to produce a tangible prod- 
uct. This complex problem-solving process in- 
volves: (1) the use of a number and variety of 
cognitive skills, such as generating ideas, 
avoiding premature judgment, and breaking 
mental "sets" in order to look at a problem in 
a new and different way; (2) the effective 
management and deployment of the various cog- 
nitive skills by what might be called a "master" 
thinking skill. For example, this Master Think- 
ing Skill would be used to determine, at a par- 
ticular point in the problem-solving sequence, 
whether it would be more fruitful to suspend 
critical judgment and give free reign to specu- 
lation and fantasy in a search for entirely new 
ideas, or whether, in contrast, it would be 
more fruitful to examine systematically and 
critically the ideas one already has; and (3) a 
high degree of involvement or dedication on the 
part of the thinker— the individual is highly en- 
grossed with the creative task and finds it 
meaningful and engaging. 

Hence the measuring instruments used in 
this study attempt to present the student with 
thinking tasks which he will find meaningful 
and engaging, and which require him to use 
and to coordinate a number of different cogni- 
tive skills. These tasks do not involve the use 
of specialized knowledge or technical skills, 
nor do they require the mastery of any particu- 
lar curriculum material. 

Because of the complex thinking which these 
tasks involve, they are not "pure" in any fac- 
torial sense and cannot be neatly classified 
as measuring thinking of any particular "type" 
(e.g., "convergent" or "divergent"). Instead, 
each test typically requires the interrelated 
use of several types of thinking. Thus the 
classification of tests which is given below 
merely indicates the relative emphasis given 
to a certain kind of thinking by the tests in 
each category; it does not mean that the tests 
in a certain category involve only one kind of 
thinking. 

A complete list and classification of the 
tests of productive thinking used in this study 
is presented in Table 1. In this classification, 
the terms "convergent" and "divergent" think- 
ing are used in the traditional way to denote 
thinking which, onthe onehand, involves pro- 
cessing information so as to "converge" upon 
a single solution to a problem or, on the other 
hand, involves the use of information as a 
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Table 1 



Classification of the Tests of Productive Thinking Used in this Study 



Name of test 


Use 


Performance indicators 


1 . 


Complex Extended Problems with an Emphasis 
on Convergent Thinking (all verbal) 


Pit problem 


pretest 


number of ideas; quality of ideas (two indices); 
achievement of solution 


Jewel problem (short form) 


pretest 


number of ideas, quality of ideas (two indices); 
achievement of solution 


Rare Coins problem 


internal test 


achievement of solution 


Jewel problem (long form) 


posttest 


number of ideas; quality of ideas (two indices); 
achievement of solution 


Black House problem 


posttest 


number of ideas; quality of ideas (two indices); 
achievement of solution; number of odd facts 
or discrepancies in information that are noted 


X-ray problem 


posttest 


number of ideas; quality of ideas (two indices); 
achievement of solution; number of relevant 
questions asked or preparatory steps taken 
before work on problem was begun 


1 . 


Complex Extended Problems with an Emphasis 
on Divergent Thinking (all verbal) 


Elevator problem 


pretest 


number of ideas 


The Deep Sea Dive 


internal test 


number of odd or puzzling facts within the 
problem that are noted; achievement of 
solution 


The Ancient City 


posttest 


number of ideas; quality of ideas (two indices); 
number of relevant questions asked or prepara- 



tory steps taken before work on the problem 
was begun 



A Visit to Karam 


posttest 


number of puzzling facts which are noted; 
intellectual persistence 




3. Brief Problems with 


an Emphasis on Convergent Thinking 
(all verbal) 


Ingenuity Items 
(Form A) 


pretest 


achievement of solution (number of solutions 
achieved; 3 possible) 


Ingenuity Items 
(Form B) 


posttest 


achievement of solution (number of solutions 
achieved: 10 possible) 


4. Brief Problems with an Emphasis on Divergent Thinking 


Squares 


pretest 

nonverbal 


number of ideas; quality of ideas (two indices) 


Truck Improvement 


pretest 

verbal 


number of ideas; quality of ideas (two indices) 
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Table 1 (cont. ) 



Itfame of Test 


Use 


Performance indicators 


Suppose People Could 
Read Minds 


pretest 

verbal 


number of ideas; number of inventions 
suggested 


A Summer Carnival 
(Form A) 


pretest 

verbal 


quality of ideas (two indices); number of 
available elements used when creating a 
product 


Circles 


posttest 

nonverbal 


number of ideas; quality of ideas 
(two indices) 


Dog Improvement 


posttest 

verbal 


number of ideas; quality of ideas 
(two indices) 


Suppose There was an 
Intelligence Pill 


posttest 

verbal 


number of ideas; number of inventions 
suggested 


A Summer Carnival 
(Form B) 


posttest 

verbal 


quality of ideas (two indices); number of 
available elements used when creating a 
product 


5. Test of the Master Thinking Skill (verbal) 


The Missing Statue 


posttest 


total number of "best" or "second best" 
alternatives to which the student gives a 
rank of one; number of occasions on which 
the student chooses the "give up" alternative 



point of departure from which one's thinking 
"diverges" into a variety of new mental prod- 
ucts. "Complex Extended Problems" in the 
classification are those which feature a gradual 
development of the problem and require the 
student to use a number of different cognitive 
skills as he proceeds. The student responds 
at several points and is presented with feed- 
back in the form of more information about the 
problem after each response opportunity. (A 
problem which uses these response- feedback 
sequences is illustrated and discussed below 
in the section on Performance Indicators and 
Scoring. ) Those tests labelled "Brief Problems" 
require the use of relatively few cognitive 
strategies and do not employ response-feedback 
sequences; instead, they present the entire 
problem to the student at once, and he then 
makes his response(s). 

All the problems are further classified as 
verbal or non-verbal, and whether they were 
used as pre- or posttests or as internal tests 
in this study. The name of each test and the 
indices of thinking proficiency on which it was 
scored are also listed. 

Performance Indicators and Scoring 

On each of the various tests, several mea- 
sures of performance were obtained; the par- 



ticular measures employed depended on the 
nature of the problem. The following example 
both illustrates the general nature of a Complex, 
Extended Problem and presents a detailed dis- 
cussion of the rationale and procedures by which 
performance was scored. 

The problem, titled The Mystery of the Old 
Black House, is presented to the student in the 
form of a brief written story. The story begins 
as a detective drives out to the country to in- 
vestigate an old black house in which gold is 
reported to be hidden. The detective finds the 
house in the late afternoon and begins his 
search, but he stops work just before sunset 
and goes to a nearby white house where he has 
dinner and spends the night. When he awakens 
the next morning, the black house has com- 
pletely disappeared, without leaving any trace 
whatsoever. The problem is to explain how the 
black house could have disappeared. Embedded 
in the story are several puzzling discrepancies 
(e.g., the detective saw the sunset through his 
bedroom window but saw the sunrise through 
the same window when he awoke the next morn- 
ing) which remain unaccounted for if one as- 
sumes that the black house must somehow have 
been torn down or moved away. However, the 
problem can be solved in a way that neatly ac- 
counts for these discrepancies by concluding 
that it was actually the detective who was 
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moved, in his sleep, to a similar white house 
some distance away. This, then, is the prin- 
cipal solution— one which meets all the require- 
ments of the problem and accounts for all the 
puzzling discrepancies as well. 

The story, containing all essentials of the 
problem, is presented to the student on the 
first page of a short booklet. On the next page, 
he is asked to write down all the ideas he has 
for explaining how the black house could have 
disappeared. He is encouraged to write as 
many ideas as possible, especially unusual 
ones. After writing his ideas, he turns the 
page and then is asked to write down any odd 
or puzzling facts he has noticed in the story 
(besides the disappearance of the black house 
itself). The following page provides feedback, 
focusing the student's attention on the several 
odd or discrepant facts in the story. Then he 
is given another opportunity to write down any 
new ideas he has for explaining how the black 
house could have disappeared. Next, a suc- 
cession of question-response-feedback units 
gradually provides the student with more and 
more information about the problem, giving him 
additional opportunities to write down any new 
ideas he has for explaining how the black 
house could have disappeared and leading him 
step— by— step toward the principal solution. 
Finally, he is given a last opportunity to write 
down ideas for explaining the disappearance of 
the black house. 

A number of indices of thinking proficiency 
can be applied to performance on such a prob- 
lem, but because of the massive amount of 
test data which this study produced, it w?s 
decided to use only those indices which were 
felt to be essential for a reasonably compre- 
hensive yet manageable analysis of perform- 
ance. In this particular problem, the following 
performance indicators were used: whether or 
not the student achieved the principal solutio_n 
to the problem; the number of odd facts or disn 
crepancies in the problem which he noticed; 
the number of ideas he wrote for explaining 
how the black house might have disappeared 
(regardless of the quality or adequacy of these 
ideas); and, finally, two indices of the .guality 
of the ideas he produced. 

The quality of an idea was judged on the 
basis of (1) the degree of imaginativeness ex- 
hibited and (2) the extent to which it accounted 
for the various facts without violating the con- 
straints of the problem. Based on previous re- 
search with this problem, a normative scale of 
quality which incorporated these two criteria 
(presented in Table 2) was prepared. Each 
idea which the student produced in the course 



of working on the problem was rated with re- 
spect to this scale. As in the case of all 
quality ratings done in this study, the ratings 
were made by trained scorers, each of whom 
first "practice rated" 50 previously obtained 
protocols and then exchanged these protocols 
with another scorer who rated them independ- 
ently. The different sets of quality ratings 
were then compared for reliability, and when 
a high degree of reliability between scorers 
was evident the regular ratings were begun. 
Because of the large number of protocols in 
this study, the task of rating the protocols 
from any given test was divided among five 
scorers. In general, each protocol was rated 
only once, so no interjudge reliabilities could 
be computed. However, a systematic program 
of rescoring sets of randomly selected protocols 
resulted in high interjudge agreement. More- 
over, because the protocols were divided 
among the scorers on a semirandom basis, the 
effect of a scoring bias by any particular scorer 
should not affect the outcomes in a systematic 
way. In addition, a "blind" scoring procedure 
was used, such that the scorers had no way of 
knowing from which experimental condition a 
given protocol had been drawn. All quality 
ratings in the study were made by the same 
five scorers. 

When a scorer had assigned a quality rating 
to each of a student's ideas in a problem, two 
indices of quality were generally recorded: the 
number of "high quality" ideas (in the case of 
the Black House problem, those receiving a 
rating of three or higher), and the quality rating. 
of the best idea which the student produced. 

This general procedure for scoring and re- 
cording the quality of ideas was used for all 
measures in which quality was scored as a per- 
formance indicator. 

Test of the Master Thinking Skill 

One problem in the test battery requires 
pecial mention because of its unique construc- 
tion and scoring. This problem, a test of the 
Master Thinking Skill, was designed specific- 
ally to measure the student's ability to manage 
and deploy the various cognitive operations 
involved in solving a complex extended prob- 
lem. The student is first presented with a 
statement of the basic problem. He then con- 
fronts a "choice point" at which he is presented 
with a list of five alternative courses of action, 
any of which might be taken at that point in the 
problem. His task is to indicate which he 
thinks is the "best" course of action, then the 
second best, and so on, by ranking these 
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Table 2 



Normative Quality Ratings for Ideas in The Mystery of the Old Black House 



Rating 


Type of ideas 


Examples 


0 


Ideas which are irrelevant, impossible, 
or contrary to fact 


The black house was never there at all; 
a magician destroyed it 


1 


Ideas which explain the apparent 
disappearance of the black house, 
but which account neither for the fact 
that no trace of the house was found 
nor for the discrepancies in the story 


The black house blew up; it was torn down 


2 


Ideas which account for both the 
apparent disappearance of the house 
and for the fact that no trace of the 
house was found, but still do not 
explain the discrepancies in the story 


It was removed by a helicopter; it was 
moved by a truck and then the tracks were 
covered; it was carefully camouflaged 
during the night 


3 


Ideas which explain the apparent 
disappearance of the black house, 
and which account for all the facts 
and discrepancies in the story, but 
can only do so by denying the reality 
of the problem 


The detective was drugged during supper, 
so he was confused when he woke up and 
onlv thought the black house had disap- 
peared; the drugs made him see things 
strangely; it was all a dream 


4 


Elegant, feasible, ideas which account 
for all the facts and events 


The principal solution: The detective was 
moved to another highly similar white 
house during the night 



alternatives from one to five. Next, he turns 
the page, and the development of the problem 
continues. After sufficient new information 
has been presented, the student encounters a 
second "choice point" with its accompanying 
list of alternative courses of action. Again, 
he indicates which of these courses he thinks 
would be best, second best, and so on. There 
is a total of six choice points, each with its 
own list of possible courses of action. 

Each list of choices always contains the 
following alternatives (in a random order, of 
course): a "best" course of action (the action 
that would be most effective at this point), a 
"second best" course of action (an action that 
is reasonable, though not as effective as the 
"best" course), a "contrary to fact" course 
of action (one that ignores or violates an al- 
ready established fact), an "appealing but 
irrelevant" action (one that is attractive but 
would not aid in solving the problem), and 
finally, in the first three choice points, a 
"jumping to conclusions" course of action 
(one which "forces" a conclusion to the prob- 
lem, but does so on the basis of insufficient 



information). For the last three choice points, 
this "jumping to conclusions" alternative is 
replaced by a "give up" alternative (in which 
the student indicates that he wants to stop 
working on the problem and simply wants to 
find out what the solution is). 

Two different scores were used as perform- 
ance indicators: the number of times (summed 
over the six choice points) that the student 
gave a rank of one to either the "best" or the 
"second best" course of action, and the number 
of times he gave a rank of one to the "give up" 
alternative presented in the last three choice 
points. The first score provides a measure of 
the Master Thinking Skill per se, while the 
second score constitutes a reciprocal measure 
of intellectual persistence (i.e., the extent to 
which the student keeps on working on the 
problem when he is given several opportunities 
to give up). Another measure of intellectual 
persistence was also obtained by the same 
general means in one other test (A Visit to 
Karam ). 
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Measures of Understanding Thinking 

In addition to measuring the student's per- 
formance on problem-solving tasks, an attempt 
was made to probe his understanding of the na- 
ture of problem solving by presenting him with 
three open-ended sentences which he was 
asked to complete. The sentences were: 
"Making mistakes in working on a problem 

is "Wild or silly ideas are 

"s and, "In order to solve problems you 

must . " 

The student's answers were rated by scorers 
on a three point scale, indicating the degree 
of insight or understanding which was shown 
by the answer. For example, in the first sen- 
tence above, a rating of zero was given to an 
answer such as "Making mistakes in working 
on a problem is bad . " A rating of one was 
given to answers such as "Making mistakes in 
working on a problem is something that happens 
even to good thinkers. " A rating of two was 
given to an answer such as "Making mistakes 
in working on a problem can show you where 
you went wrong and lead you to new ideas. " 

Each of the three sentences was scored as 
a separate variable and is referred to as 
Sentence I, II, or III, respectively, in the 
analysis. 

EXPERIMENTAL DESIGN AND STATISTICAL ANALYSES 

The basic analyses for this experiment 
were done with a 2 X 2 X 2 factorial design. 

The factors represented were: Instruction 
(Treatment vs. Control); Environment (Facilita- 
tive vs. Nonfacilitative); and Sex. There were 
11 classrooms within each of the cells of this 



design, and standard analysis of variance pro- 
cedures were used for the analyses. 

Because the investigators also wished to 
look at the relationships between IQ and the 
Treatment and Environment factors, a second- 
ary analysis was performed, using a 3 X 2 X 2 
factorial design. Three levels of IQ were 
represented in this design— high (above 115), 
middle (101 through 115), and low (100 and 
below). The other two factors were Treatment- 
Control and Facilitative-Nonfacilitative Environ- 
ment. After all subjects had been blocked into 
three groups on the basis of IQ, a random 
sample of 20 was selected at each IQ level 
within each Treatment X Environment combina- 
tion in the design. Thus, a total of 240 sub- 
jects was used for this secondary analysis. 

(In order to obtain such a sample, it was nec- 
essary to ignore classifications according to 
sex and classroom units involved in the experi- 
ment. ) 

For bo;h the main analysis (the 2X2X2 
factorial design) and the secondary analysis 
(the 3X2X2 factorial design involving IQ), 
separate analyses of variance were performed 
on each of five stratification variables— 
socioeconomic status, overall mean score on 
the Stanford Achievement Battery, IQ, perform- 
ance on the Sequential Test of Educational 
Progress (Social Studies, Form 4A) when given 
as a pretest, and scores on this test when 
given as a posttest (using Form 4B). Similar 
analyses were performed on all pretest, in- 
ternal test, and posttest measures. 

In the second design, the analysis was 
performed using scores of individual students 
as the basic unit, in contrast to the main de- 
sign where the classroom mean score for each 
variable was the unit of analysis. 
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RESULTS 



Tables 3, 4 and 5 present the means and 
standard deviations of all the variables ana- 
lyzed in this experiment. Values for the strati- 
fication variables are in Table 3, for the pre- 
test measures in Table 4, and for the internal 
and posttest measures in Table 5. 

The results of the analyses of variance for 
the stratification variables are given in Table 
6. Note that there were no significant Treat- 
ment-Control differences on any of these 
variables. The Environment factor, on the 
other hand, represented a significant source 
of variance -on all five of these variables. 
Hence the supervisor's ratings of what con- 
stituted a "facilitative" classroom environ- 
ment reflected, to some extent, the calibre 
of students in the class being rated. However, 



with the exception of socioeconomic status, 
the mean differences between Facilitative and 
Nonfacilitativ.e classes on these stratification 
variables were fairly small, although statis- 
tically significant. Means for the Treatment- 
Control and Facilitative-Nonfacilitative 
Environment groups on the stratification vari- 
ables are presented in Table 7. 

The data in Table 6 also indicate significant 
sex differences in achievement and IQ. These 
differences arise from the fact that girls are 
superior to boys on these two variables (Table 
8). (Although differences on the two forms of 
the STEP test were not significant, they were 
in the same direction.) These differences are 
consistent with other findings concerning sex 
differences in aptitude and achievement at 
this level of development. 



Table 3 

Means and Standard Deviations of Stratification Variables 



Standard 



Variable 


Mean 


Deviation 


Socioeconomic status 


2.98 


1.49 


Stanford achievement 


5.09 


.65 


Kuhlmann-Anderson IQ 


107.39 


7.68 


STEP Test \Pretest: Form 4A) 


26.44 


3.22 


STEP Test (Posttest: Form 4B) 


25.48 


3. 15 



Table 4 



Means and Standard Deviations of Pretest Variables 



Variable 



Mean 



Standard 

Deviation 



Extended problems in convergent thinking 



Pit problem: 

Achievement of solution 
Number of ideas 
'Number high quality ideas 
Quality rating, best idea 

Jewel problem: 

Achievement of solution 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 

a 

Brief problem in convergent thinking 

Ingenuity test: 

Achievement of solution 

Extended problem in divergent thinking 

Elevator problem: 

Number of ideas 

Brief problems in divergent thinking 

Squares problem (nonverbal): 

Number of ideas 
Number high quality ideas 
Quality reding, best idea 

Truck improvement problem (verbal): 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 

Mind-reading problem (verbal): 

Number of ideas 
Inventiveness 

Summer carnival problem (verbal): 
Number high quality ideas 
Quality rating, best idea 
Ability to synthesize elements 



.42 
3.63 
1. 08 
3. 02 



. 06 
3.53 
.37 
1. 10 



.48 



2.92 



8.45 

1.92 

1.65 



8. 04 
1.42 
1.64 



3.94 

.20 



.40 

6.10 

2.10 



.20 

.74 

.37 

.43 



.09 
1. 06 
.21 
. 35 



.26 



1. 08 



2.49 

.83 

.26 



2. 30 
.62 
.22 



1.43 

.25 



.29 

.30 

.59 



a Problems in this group are all verbal. 
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Table 5 



Means and Standard Deviations of Internal and Posttest Variables 



Variable 


Mean 


Standard 

Deviation 


a 

Internal test measures 






Rare coins problem (extended, convergent): 
Achievement of solution 


.21 


. 19 


Deep Sea Dive problem (extended, divergent): 
Achievement of solution 
Sensitivity to incongruous information 


.27 

1.46 


.19 

.57 


cl 

Extended problems in convergent thinking 






Jewel problem: 

Achievement of solution 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 


. 51 
6. 18 
1.29 
2.23 


.25 

1.57 

.35 

.45 


House problem: 

Achievement of solution 

Number of ideas 

Number high quality ideas 

Quality rating, best idea 

Sensitivity to incongruous information 


.65 
4. 18 
1.17 
4.09 
.53 


.20 

.79 

.38 

.58 

.36 


X-ray problem: 

Achievement of solution 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 
Questioning and clarifying ability 


. 15 
2.79 
. 82 
2. 58 
. 63 


.11 

.83 

.41 

.45 

.44 


Brief problem in convergent thinking 






Ingenuity: 

Achievement of solution 


3.27 


.99 


g 

Extended problems in divergent thinking 






Ancient city problem: 

Number of ideas 
Number high quality ideas 
Quality rating, best idea 
Questioning and clarifying ability 


4.79 
.35 
1. 00 
.87 


1. 54 
.20 
.33 
.58 


A Visit to Karam: 

Sensitivity to puzzling facts 
Intellectual persistence 


.74 

2.46 


.78 

.84 



a Problems in this group are all verbal# 

(Continued on next page) 
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Table 5 (cont'd) 



Variable 



Brief problems in divergent thinking 

Circles (nonverbal): 

Number of ideas 
Number high quality ideas 
Quality rating, best idea 

Dog improvement (verbal): 

Number of ideas 
Number high quality ideas 
Quality rating, best idea 

Intelligence pill (verbal): 

Number of ideas 
Inventiveness 

Summer carnival (verbal): 

Number high quality ideas 
Quality rating, best idea 
Ability to synthesize elements 

Special tests 

Statue problem: 

Effectiveness of Master Thinking Skill 
Intellectual persistence 

Understanding thinking: 

Sentence I 
Sentence II 
Sentence III 

a Problems in this group are all verbal. 



Mean 



Standard 

Deviation 



14. 08 


2. 52 


2. 08 


. 75 


1.76 


. 18 



7.92 


2. 18 


1.19 


.60 


1.54 


.24 



2.81 


r- 

00 

• 


. 06 


. 09 



.47 


.26 


1.16 


.32 


1.80 


.61 



3.53 


. 66 


1.59 


O 

00 

• 



. 66 


.37 


1. 00 


.28 


1.24 


.23 



Table 6 

Summary of Analyses of Variance for Stratification Variables 

(df = 1, 80) 



Variable 


Treatment 


Environment 


Sex 




MS 


F 


MS 


F 


MS 




F 


Socioeconomic status 


. 05 


<1 


24. 04 


11. 58** 


. 00 


< 


1 


Stanford achievement 


. 01 


<1 


2. 09 


5. 64* 


3. 01 


7. 


81** 


Kuhlmann-Anderson IQ 


.75 


<1 


484. 02 


9.37** 


454. 70 


8. 


8 1 ** 


STEP Pretest 


. 17 


<1 


47.39 


4. 55* 


6.60 


< 


1 


STEP Posttest 


6 . 36 


<1 


42.62 


4. 30* 


3. 95 


< 


1 


*p < . 05. 

**p < .01. 
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Table 7 



Mean Level of Performance of Students in Treatment 
and Control Groups and in Facilitative and Nonfacilitative 
Environments: Stratification Variables 



Variable 


Group 


Environment 


Treatment 


Control 


Facilitative 


Nonfacilitative 


Socioeconomic status** 


2. 95 


3. 00 


2.45 


3. 50 


Stanford achievement* 


5. 11 


5. 09 


5. 25 


4. 93 


Kuhlmann-Anderson IQ** 


107.49 


107. 31 


109.74 


105. 05 


STEP Pretest* 


26.40 


26. 49 


27. 18 


25.71 


STEP Posttest* 


25.22 


25. 77 


26. 18 


24. 79 


*' 'Environment" difference significant at p < 
**" Environment" difference significant at p < 


• • 
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Table 8 

Mean Level of Performance of Males and Females: 
Stratification Variables 



Variable 


Males 


Females 


Socioeconomic status 


2. 98 


2. 98 


Stanford achievement* 


4.91 


5.28 


K-A IQ* 


105. 12 


109.67 


STEP Pretest 


26. 17 


26. 72 


STEP Posttest 


25.27 


25. 70 



^Difference significant at p < . 01 



PRETEST RESULTS 

In Table 9 is a summary of the analyses of 
the 21 pretest variables. In the 21 analyses, 
there was only one Treatment effect significant 
atthe.05 level. It seems likely that this sig- 
nificant F is simply an artifact of the number of 
analyses carried out. The means for treatment 
and control group's are presented in Table 10. 

The Environment factor produced significant 
differences on five pretest measures, all of 
which concerned the two brief problems in di- 
vergent thinking— the nonverbal Squares prob- 
lem and the verbal Truck problem. Inspection 
of the means in Table 11 reveals that these 
differences are all in the same direction, and 



all reflect the superior performance of students 
in the Facilitative environment. 

Tables 12 and 13 present the means for 
males and females on the pretest measures. 
Except for the Pit problem, where males were 
superior to females on three of the four per- 
formance indicators, females performed better 
than males for all measures on which statis- 
tically significant sex differences were found. 

These, then, were the differences which 
existed among fifth-grade children on various 
measures of productive thinking before any 
special training was administered. Let us 
turn now to the results obtained after Series 
One of The Productive Thinking Program had 
been administered* 
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Table 9 



Summary of Analyses of Variance of Pretest Variables 
(1 and 80 degrees of freedom) 



Main effects 

Treatment Environment Sex 

Variables MS F MS F MS F 



g 

Extended problems in convergent thinking 


Pit problem: 


Achievement of solution 


. 00 


Number of ideas 


1.34 


Number high quality ideas 


. 06 


Quality rating, best idea 


. 02 


Jewel problem: 


Achievement of solution 


. 04 


Number of ideas 


.54 


Number high quality ideas 


. 03 


Quality rating, best idea 


.38 



g 

Brief problem in convergent thinking 



Ingenuity test: . 

Achievement of solution . 00 

g 

Extended problem in divergent thinking 
Elevator problem: 

Number of ideas • 57 

Brief problems in divergent thinking 

Squares problem (nonverbal): 

Number of ideas 22.76 

Number high quality ideas • 26 

Quality rating, best idea . 00 

Truck improvement problem (verbal): 

Number of ideas . 00 

Number high quality ideas . 00 

Quality rating, best idea . 03 

Mind-reading (verbal): 

Number of ideas . 05 

Inventiveness . 00 

Summer carnival (verbal): 

Number high quality ideas . 01 

Quality rating, best idea . 01 

Ability to synthesize elements . 05 

a Problems in this group are all verbal. 



< 1 


. 00 


< 1 


’ .41 


11. 59** 


2.59 


1. 93 


3.74 


2.21 


4. 27* 


< 1 


. 07 


< 1 


1.18 


9. 31** 


< 1 


. 00 


< 1 


2. 16 


13. 07** 


5 . 46 * 


. 00 


< 1 


. 01 


1. 78 


< 1 


2.42 


2.11 


.49 


< 1 


< 1 


.11 


< 1 


. 07 


< 1 


3.17 


. 11 


< 1 


. 07 


< 1 



< 1 


.25 


3.68 


.22 


3. 34 


< 1 


. 07 


< 1 


3. 87 


3. 19 


3.92 


23.66 


4. 08* 


19.64 


3. 38 


< 1 


5.73 


8. 90** 


. 80 


1.24 


< 1 


.32 


4. 70* 


. 15 


2.21 


< 1 


.98 


< 1 


4.91 


< 1 


< 1 


2. 04 


5.37* 


. 24 


< 1 


< 1 


.27 


5. 56* 


. 02 


< 1 . 


< 1 


.07 


< 1 


15, 54 


8. 38** 


< 1 


. 01 


< 1 


. 15 


2.45 


< 1 


. 03 


< 1 


.47 


5.67* 


< 1 


. 01 


< 1 


.47 


5. 10* 


< 1 


. 01 


< 1 


5. 77 


19. 11** 



1 _ 

Number of solutions out of a possible three. 

*p < . 05 

**p < . 01 
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Table 10 

Mean Level of Performance of Treatment and Control Groups: 


Pretest Variables 


Variable 


Treatment 


Control 


Pit problem: 


Achievement of solution 


.43 


. 41 


Number of ideas 


3.75 


3. 51 


Number high quality ideas 


1. 05 


1.10 


Quality rating, best idea 


3. 00 


3. 03 


Jewel problem: 


Achievement of solution 


. 08 


. 04 


Number of ideas 


3.61 


3. 45 


Number high quality ideas 


.39 


.36 


Quality rating, best idea 


1.16 


1. 04 


Ingenuity test: 


Achievement of solution 


.48 


.48 


Elevator problem: 


Number of ideas 


3. 00 


2. 84 


Squares problem: 


Number of ideas 


8.96 


7. 94 


Number high quality ideas 


1.98 


1. 87 


Quality rating, best. idea 


1.66 


1 . 64 


Truck improvement problem: 


Number of ideas 


8. 03 


8 . 04 


Number high quality ideas 


1.43 


1.42 


Quality rating, best ideas 


1.65 


1.62 


Mind-reading problem: 


Number of ideas 


3. 97 


3. 92 


Inventiveness 


.20 


. 21 


Summer carnival: 


Number high quality ideas 


.42 


. 39 


Quality rating, best idea 


1.11 


1. 09 


Ability to synthesize elements 


2.12 


2. 08 



Table 11 



Comparison of Ideas Generated by Students 
in Facilitative (F) and Nonfacilitative (N) Environments: Pretest Variables 





Fluencv 


No. hiah 


aualitv 


Oualitv 


of best 


Problem 


F 


N 


F 


N 


F 


N 


Pit 


3.48 


3. 78 


1. 05 


1.11 


3. 01 


3. 02 


Jewel 


3. 36 


3. 70 


.40 


.35 


1.13 


1. 06 


Elevator 


2. 94 


2.89 










Squares 

Truck 


8. 97 * 

8. 14 


7. 93* 
7. 93 


2. 18** 
1. 58* 


1.67** 

1.27* 


1.71* 

1.69* 


1. 59* 
1. 58* 


, No. of Ideas 


3. 97 


3.91 


— 


— 




” 


Mind: 

Inventiveness 


.21 


• 19 a 


- 


- 


— 




Carnival 


2. 1 1 3 


2. 09 


.42 


.38 


1. 01 


l.H 



a Not a measure of fluency. This entry represents the number of available elements used when 
creating a "product. " 



^Difference significant at p < . 05. 
❖❖Difference significant at p < . 01 . 



Table 12 



Mean Level of Achievement of Solution for Males and Females: Pretest Variables 



Test 


Males 


Females 


Pit* 

Jewel 

Ingenuity 


.49 
. 07 
. 53 


.35 
. 05 
.43 



❖Difference significant at p < . 01 



Table 13 



Comparison of Ideas Generated by Males (M) and Females (F): Pretest Variables 



mi 


Fluencv 


No. hiqh 


aualitv 


Oualitv 


of Best 




M 


F 


M 


F 


M 


F 


Pit 


3.47* 


3. 79* 


1.19** 


,96 ** 


3. 17** 


2. 86** 


Jewel 


3.46 


3.61 


.39 


.36 


1.13 


1. 07 


Elevator 


2.71 


3. 12 


mm 


“ 




1.69 
1. 62 


Squares 

Truck 


7.97 
7. 80 


8.92 

8.28 


1. 83 
1.48 


2. 02 
1.37 


1.61 

1.65 


No. of Ideas 


3 . 52** 


4. 36** 


— 


- 


mm 




^ in : Inventiveness 


. 16 


.24 


- 


— 




“ 


Carnival 


1. 84 a ** 


2. 36 a ** 


. 33* 


.48* 


1. 03* 


1.17* 



a Not a measure of fluency. This entry represents the number of available elements used when 



creating a "product." 

^Difference significant at p < . 05. 
**Difference significant at p < . 01. 
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Table 14 



Summary of Analyses of Variance: 

Main Effects for Internal and Posttest Variables 
(1 df for effects) 







Source 




Variable 


Treatment 


Environment 


Sex 


MS F 


MS F 


MS F 



Q 

Internal Measures 
Rare coins problem: 



Achievement of solution 


. 02 


< 1 


. 07 


2 . 15 


. 00 


< 1 


Deep sea dive problem: 


Achievement of solution 


. 05 


1.46 


. 00 


< 1 


. 00 


< 1 


Sensitivity to incongruous 


2. 25 


7.41** 


. 00 


< 1 


1.25 


4. 11* 


information 


a 

Extended problems in convergent thinking 












Q 

Jewel problem: 


Achievement of solution 


. 11 


1.61 


. 01 


< 1 


. 01 


< 1 


Number of ideas 


13.46 


6 . 59* 


. 05 


< 1 


4.78 


2.34 


Number high quality ideas 


. 25 


2. 37 


. 02 


< 1 


.37 


3. 50 


Quality rating, best idea 


.62 


2 . 80 


. 16 


< 1 


.04 


< 1 


House problem: 


Achievement of solution 


.23 


6.47* 


. 11 


2 . 98 


. 04 


1. 15 


Number of ideas 


.54 


< 1 


. 14 


< 1 


1.61 


2. 52 


Number high quality ideas 


2 . 05 


1 6 . 99** 


. 03 


< 1 


.57 


4. 71* 


Quality rating, best idea 


1. 84 


5. 98* 


1. 13 


3. 67 


.52 


1. 68 


Sensitivity to incongruous 


1. 50 





. 06 


< 1 


.67 


7. 00** 


information 
X-ray problem: 


Achievement of solution 


. 16 


6.20* 


. 03 


1 . 04 


. 00 


< 1 


Number of idea-s 


. 00 


< 1 


. 01 


< 1 


1. 05 


1.45 


Number high quality ideas 


.45 


2.73 


. 01 


< 1 


. 01 


< 1 


Quality rating, best idea 


.99 


5.22* 


. 00 


< 1 


. 02 


< 1 


Sensitivity to incongruous 


. 50 


2. 83 


. 00 


< 1 


1.70 


9. 59** 


information 



g 

Brief problem in convergent thinking 



Ingenuity:^ 



Achievement of solution 


45 


< 


1 


4. 84 


4. 


O' 

co 


.92 


< 


1 


g 

Extended problems in divergent thinking 




















q 

Ancient city problem: 




















Number of ideas 22. 


66 


10 . 


9 7 


1. 59 


< 


1 


.11 


< 


1 


Number high quality ideas 


01 


< 


1 


. 00 


< 


1 


. 08 


1. 


88 


Quality rating, best idea 


16 


1. 


39 


. 00 


< 


1 


. 18 


1. 


, 60 


Questioning and clarifying ability 


06 


< 


1 


.49 


1. 


47 


1.15 


3. 


44 



a Problems in this group are all verbal. 

^Total number of solutions out of a possible ten. 
c 36 df for error. 

^80 df for error. 

*p < . 05 

** p <.01 
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Table 14 (cont'd) 



Source 



Treatment Environment Sex 



Variable 


MS 


F 


MS 


F 


MS 


F 


q 

A Visit to Karam: 


Sensitivity to puzzling facts 


1.32 


2. 12 


. 00 


< 1 


. 00 


< 1 


Intellectual persistence 


2.74 


4. 15* * 


1.78 


2.69 


.23 


< 1 


Brief problems in divergent thinking 


Circles (nonverbal): 
Number of ideas 


10. 00 


1.68 


5. 02 


< 1 


55.63 


9. 37** 
< 1 
< 1 


Number high quality ideas 


. 08 


< 1 


„ 53 


< 1 


.37 


Quality rating, best idea 


. 04 


1.39 


. 01 


< 1 


. 00 


Dog improvement (verbal): 


Number of ideas 


10.98 


2.72 


6.20 


1.53 


57. 05 


14.13** 


Number high quality ideas 


.34 


1. 03 


.57 


1.71 


1.52 


4.55* 


Quality rating, best idea 


. 00 


< 1 


. 08 


1.38 


.34 


1. 09 


Intelligence pill (verbal): 


Number of ideas 


.54 


< 1 


.79 


1.20 


6.61 


9.99** 


Inventiveness 


.38 


4.33* 


. 00 


< 1 


. 00 


< 1 


Summer carnival (verbal): 


Number high quality ideas 


. 02 


< 1 


. 02 


< 1 


.98 


16.97** 


Quality rating, best idea 


. 01 


< 1 


. 01 


< 1 


1.70 


20. 49** 


Ability to synthesize elements 


.22 


< 1 


. 01 


< 1 


4.99 


15.41** 



Special tests 
Statue problem: 

Effectiveness of Master Thinking 



Skill 1 . 97 

Intellectual persistence . 04 

q 

Understanding thinking: 

Sentence I .68 

Sentence II .13 

Sentence III . 08 



5. 06* 


. 14 


< 1 


1.59 


4. 08 


< 1 


.09 


< 1 


1.50 


< 1 


6.79* 


. 33 


3.27 


.81 


8. 12** 


1.58 


. 01 


< 1 


.28 


3.42 


1. 56 


. 06 


1.17 


. 18 


3.45 



a Problems in this group are all verbal. 

^Total number of solutions out of a possible ten. 



c 36 df for error. 
^80 df for error. 

*p < . 05. 

**p < . 01. 



INTERNAL AND POSTTEST RESULTS 

Table 14 summarizes the results of the 
analyses of variance of all internal and post- 
test measures. Of particular interest are the 
Treatment-Control differences, thirteen of 
which are statistically significant (as opposed 
to only one such difference that reached sig- 
nificance on the pretest measures). Table 15 
presents the means for the Treatment and 
Control groups on incidence of achievement 
of solution for problems which have an empha- 
sis on convergent thinking (the type of problem 



which permits only a small set of "best" solu- 
tions). 2 Two of the six measures show sig- 
nificant differences, and on five of the six 
measures performance of the Treatment group 
exceeds that of the Controls. 

The sixth measure, involving the X-ray 
problem, requires special comment since this 
problem was the only one in the entire posttest 



2 The Deep Sea Dive problem, although it em- 
phasizes divergent thinking, was included in 
this Table because it, too, permits only a 
small set of "best" solutions. 



0 
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Table 15 



Mean Level of Achievement of Solution for 
Treatment and Control Groups: 
Internal and Posttest Variables 



Problem 


Treatment 


Control 


Rare coins 


.22 


. 19 


Deep sea dive 


.29 


.24 


Jewel 


• 56 


.45 


House* 


.70 


.59 


X-ray* 


.11 


. 19 


Ingenuity 


3.34 


3.20 



^Difference significant at p < .05. 



battery on which the Control group consistently 
outperformed the Treatment group by a statis- 
tically significant margin. This unexpected 
reversal of the general pattern of Treatment 
group superiority appears to be an artifact of 
this particular problem, most likely due to the 
fact that the task was so difficult (only 15% of 
all students ever achieved solution) that an 
outstanding performance by relatively few Con- 
trol students could produce significant differ- 
ence in favor of that group. 

On measures dealing with the number and 
quality of ideas (Table 16), six significant 
Treatment-Control differences were found. Five 
of these differences favored the Treatment group, 
while the sixth (again involving the X-ray prob- 



lem) favored the Controls. The means for 
Treatment-Control performance on all remaining 
internal and posttest measures are presented in 
Table 17, and all significant differences in 
this table were the result of superior perform- 
ance by the Treatment group. 

In brief, then, Series One of The Productive 
Thinking Program produced significant Treatment- 
Control differences on one-third (13 out of 40) 
of the internal and posttest measures, when only 
ono significant difference (out of 21 measures) 
existed before training began. Eleven of the 
significant posttest differences indicated supe- 
rior performance by the Treatment group, while 
the two differences which did not both involved 
the maverick X-ray problem. 

Turning now to differences resulting from the 
two types of classroom environment, Table 14 
shows that only one such difference (out of the 
40 posttest analyses) was statistically signifi- 
cant. This is a considerable contrast to the 
pretest results in which five significant dif- 
ferences (out of 21 analyses) were found, all 
of which indicated the superiority of a Facilitative 
environment. The notable lack of posttest ef- 
fects may partly be accounted for by the pres- 
ence of several significant Treatment X Environ- 
ment interactions (see below), all of which 
indicate that treatment was more effective with 
students in Nonfacilitative environments, thus 
reducing the initial differences between Facili- 
tative and Nonfacilitative groups. Several addi- 
tional Treatment X Environment interactions, 



Table 16 

Comparison of Ideas Generated by Students in Treatment (T) and Control (C) Groups: 

Posttest Variables 



Problem 


Fluencv 
T C 


No. high gualitv 
T C 


Oualitv of best 
T C 


Jewel 


6.77* 


5. 58* 


1.37 


1.21 


2.35 


2.11 


House 


4.26 


4. 10 


1. 32** 


1. 01** 


4.23* 


3. 94* 


X-ray 


2.79 


2.79 


.75 


.89 


2.48* 


2.69* 


Ancient city 


5.22** 


4. 06** 


. 36 


.34 


1. 06 


.95 


Circles 


14.42 


13.75 


2. 05 


2.11 


1.74 


1 0 7 8 


Dog 


8. 28 


7. 57 


1.26 


1.13 


1.54 


1*53 


No. of ideas 


2.73 


2. 88 


— 


*P* 


mm 




Pill* 

' Inventiveness 


. 08* 


. 03* 


- 


- 




mm 


Carnival 


1.75 3 


1.85 a 


.45 


.48 


1.15 


1.17 



a Not a measure of fluency. This entry represents the number of available elements used when 
creating a "product." 

^'Difference significant at p < . 05. 

❖❖Difference significant at p < . 01 . 
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Table 17 



Means for Treatment and Control Groups on Remaining Internal and Posttest Measures 



Variable 


Treatment 


Control 


Deep sea dive problem: 






Sensitivity to incongruous information 


1.63* 


1 . 29** 


House problem: 






Sensitivity to incongruous information 


. 66* 


. 40** 


X-ray problem: 






Sensitivity to incongruous information 


. 55 


.71 


Ancient city problem: 






Questioning and clarifying ability 


.90 


. 84 


A Visit to Karam: 






Sensitivity to’puzzling facts 


.90 


. 58 


Intellectual persistence 


2.7 0* 


2.22* 


Statue problem: 






Effectiveness of Master Thinking Skill 


3.73* 


3.32* 


Intellectual persistence 


1.56 


1. 61 


Insight and understanding of thinking: 






Sentence I 


.7 8* 


. 53* 


Sentence II 


1. 05 


. 95 


Sentence III 


1.28 


1.20 



^Difference significant at p < .05. 
>:« ^Difference significant at p < . 01. 



though they failed to reach significance, also 
showed this pattern. 

Each of the significant sex differences re- 
ported in Table 14 (12 in 40 analyses) reflects 
the superior performance of females. Seven 
of the 12 differences occurred on variables 
dealing with the production of ideas (see Table 
18), three' differences reflected the greater sen- 
sitivit.ytof females to discrepant or puzzling 
information embedded in the tests, and the 
remaining two differences occurred on a variety 
of measures. With respect to the verbal fluency 
measures, the data are consistent with the 
numerous findings that girls are verbally more 
proficient than boys. 

In the analyses of all these measures — pre- 
test, internal test, and posttest — there was a 
total of seven significant interactions, all in 
the internal and posttests. Two of these, 
three-way interactions of Treatment X Environ- 
ment X Sex, were uninterpretable. The other 
five were Treatment X Environment interactions, 
for which the means are presented in Table 19. 
The interactions for these five measures all 
have the same pattern: within the Treatment 
condition those students in Nonfacilitative 
environments performed better than students in 



Facilitative environments, while for the Control 
group, students in the Facilitative environments 
performed better than those in the Nonfacilita- 
tive environments. Thus, it would seem that 
one important effect of the training materials 
was to enable students from less favorable 
classroom climates to overcome the detrimental 
effects that such environments would otherwise 
have produced on their performance. This helps 
explain the fact (noted earlier) that classroom 
environment was a significant source of variance 
on 5 out of 21 pretest measures, but on only 1 
out of 40 posttest measures. 

EFFECTS OF IQ 

The second major analysis, carried out with 
the 3X2X2 design, involved three levels of 
IQ (above 115, 101-115, and 100 or below), the 
two Treatment conditions, and the two types of 
Environment. The purpose of this analysis was 
to determine the effect of IQ on performance, 
both as a main effect and in its interactions 
with Treatment or Environment. Since the main 
effects of the Treatment and Environment factors 
have already been discussed, they will not be 
considered in connection with this analysis. 
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Table 18 



Comparison of Ideas Generated by Males (M) and Females (F): Posttest Variables 



Problem 


Fluencv 


No. hicrh cmalitv 


Quality of best 


M 


F 


M 


F 


M 


F 


Jewel 


5. 84 


6. 51 


1.20 


1.38 


2. 20 


2. 27 


House 


4. 04 


4.31 


1. 08* 


1.25* 


4. 01 


4. 16 


X-ray 


2. 68 


2.90 


.81 


.83 


2. 60 


2. 57 


Ancient city 


4.74 


4.84 


.31 


.39 


.94 


1. 06 


Circles 


13. 29** 


14. 88** 


2. 14 


2. 01 


1.76 


1.76 


Dog 


7. 12** 


8.73** 


1. 06* 


1.32* 


1.47 


1.60 


Pill 


2. 53** 


3. 08** 


- 


— 


— 


— 


Q 

Carnival 


1. 56** 


2. 03** 


. 36** 


. 57** 


1. 02** 


1.30** 


"Fluency" means reflect the number of elements combined to generate a 
^Difference significant at p < .05. 

❖^Difference significant at p < . 01. 


"product. " 







Table 19 






Means for Significant Treatment X Environment Interactions 






(1 df for Interaction) 






Variable 


df for 




Environment 


error 




Facilitative 


Nonfacilitative 


Rare coins problem (internal test): 


80 








Achievement of solution 




Treatment 


.23 


.22 






Control 


.28 


.12 


Jewel problem (posttest): 


36 








Number of ideas 




Treatment 


6.25 


7.21 






Control 


6.03 


5.21 


Dog problem (posttest): 


80 








Number of ideas 




Treatment 


8.10 


8.46 






Control 


8.28 


6. 86 


Number high quality ideas 




Treatment 


1.20 


1.31 






Control 


1.34 


.92 


House problem (posttest): 


80 








Sensitivity to incongruous information 


Treatment 


.54 


.79 






Control 


.48 


.33 



Table 20 presents the results of this IQ 
analysis for the stratification variables. With 
the exception of socioeconomic status, perform- 
ance on each of these variables was signifi- 
cantly related to IQ (p < . 01). Table 21 indi- 
cates that the mean differences for these 
variables are all in the predicted direction- 
high IQ students perform better on the Stanford 



and STEP tests than do middle IQ students, who 
in turn perform better than those with low IQ. 

With respect to the pretest measures, Table 
22 indicates that IQ represented a significant 
source of variance for 17 of the 21 measures 
.(15 of these beyond the . 01 level). The mean 
levels for achievement of solution to the three 
convergent thinking problems used in the pretest 



0 



23 



r 



1 



are presented in Table 23. In each of these 
problems, the high IQ group outperformed the 
others, and the middle IQ group, with one 
small reversal, was consistently superior to 



the low IQ group. For measures of the quantity 
and quality of ideas produced on the various 
tests, this same rank-order relation between 
IQ and performance was found (Table 24). 



Table 20 



Summary of Analyses of IQ as a Main Effect: 
Stratifying Variables 
(df for Effect = 2) 





df for 
Error 


IQ 




Variable 


MS 


F 


Socioeconomic status 
Stanford achievement 
STEP Pretest 
STEP Posttest 


228 

225 

215 

215 


.11 

108.66 
922. 08 
1049.46 


< 1 

154. 58* 
46. 34* 
55.66* 


*p < . 01 


Table 21 






Means of IQ Groups on Stratifying Variables 








IQ 




Variable 


High 


Middle 


Low 


Socioeconomic status 
Stanford achievement* 
STEP Pretest* 

STEP Posttest* 


2.84 

6.32 

29.73 

29.18 


2.88 
5. 13 
27.79. 
26.75 


2.91 
3.96 
22. 92 
21.78 


^‘Differences significant at p < . 01 . 
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Table 22 



Summary of Analyses of Main Effect of IQ: 
Pretest Variables 
(df for Effect = 2) 





df for 


IQ effect 




Variable 


error 


MS 


F 


g 

Extended problems in convergent thinking 








Pit problem: 

Achievement of solution 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 


221 


1.32 

5.68 

5.15 

9.62 


5. 67** 
2.40 
5.48** 
9. 16** 


Jewel problem: 

Achievement of solution 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 


225 


.46 

41.25 

2.47 

9.51 


6.92** 
10.82** 
7 . 1 0** 
12. 52** 


g 

Brief problem in convergent thinking 






t 


Ingenuity test: ^ 

Achievement of solution" 

g 

Extended problem in divergent thinking 


224 


6. 68 


17.54** 


Elevator problem: 
Number of ideas 


221 


• 

38.39 


10.80** 


Brief problems in divergent thinking 








Squares (nonverbal): 

Number of ideas 
Number high quality ideas 
Quality rating, best idea 


221 


489.51 

47.83 

3.62 


26.76** 

15.92** 

12.70** 


Truck improvement (verbal) 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 


221 


226.61 

5.34 

.46 


13.52** 
2. 52 
1.67 


Mind-reading (verbal): 
Number of ideas 
Inventiveness 


221 


142.75 

.68 


26.39** 

2.38 


Summer carnival (verbal): 

Number high quality ideas 
Quality rating, best idea 
Ability to synthesize elements 


225 


1.17 

2.24 

7.62 


3.99* 

4.31* 

5.26** 



a Problems in this group are all verbal. 

^Total number of solutions out of a po^iblQ three. 



*p < . 05 

**p < . 01. 
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Table 23 



Mean Level of Achievement of Solution for 
IQ Groups: Pretest Variables 



Problem 




IQ 




High 


Middle 


Low 


Pit* 


.55 


.41 


. 28 


Jewel* 


. 16 


. 03 


. 04 


Ingenuity* 


• 

00 

o 


.46 


. 30 



^Differences significant at p < .01 



Turning now to the effect of IQ on the post- 
test measures (IQ effects on the three internal 
test measures were not analyzed), Table 25 
shows significant differences on 27 of the 38 
measures (19 of these beyond the .01 level). 

For problems emphasizing convergent thinking, 
significant differences occurred on 13 of the 
16 measures; for problems emphasizing diverg- 
ent thinking, significant differences were found 
on 13 of the 17 measures; and on 1 of the re- 
maining 5 measures a significant difference 
was found. These means are presented in 



Tables 26, 27 and 28; in all cases, the high 
and middle IQ groups performed substantially 
better than the low IQ group. However, per- 
formance of the high and middle IQ groups on 
these posttest measures was much more alike 
(relative to the low IQ group) than it had been 
in the pretest. The absence of any systematic 
IQ X Treatment interaction (see below) sug- 
gests that this change is not of sufficient 
magnitude to be considered an important result 
of exposure to the program. As in the pretest, 
there was a general rank-order relation between 
IQ and performance on most of the posttest 
measures. 

A total of nine significant interactions oc- 
curred in this second analysis, four involving 
pretest measures and the other five involving 
the posttest. The means for these significant 
interactions are presented in Tables 29 and 30. 
No consistent pattern appears to be present in 
these interactions. (Two posttest interactions 
are omitted, one because it is an IQ X Treat- 
ment x Environment interaction which yields 
no meaningful interpretation, the other because 
it is a Treatment X Environment interaction 
which does not involve IQ and about which 
information is best obtained from the first 
analysis. ) 



Table 24 



Comparison of Ideas Generated by IQ Groups: Pretest Variables 



Problem 




Fluencv 




No. 


hicrh quality 


Quality of best 


H 


M 


L 


H 


M 


L 


H 


M 


L 


Pit 


3.90 


3.88 


3.41 


1.35* 


1. 09* 


. 83* 


3.. 29* 


3. 05* 


2. 59* 


Jewel 


4. 31* 


3.49* 


2.86* 


. 58* 


. 33* 


.23* 


1.48* 


1. 08* 


.78* 


Elevator 


3.64* 


3.24* 


2 . 27 * 


— 


— 










Squares 


10.97* 


8.30* 


5. 87* 


2.62* 


1.76* 


1. 04* 


1.86* 


1.69* 


1.42* 


Truck 


9.21* 


8.73* 


6.04* 


1.60 


1.41 


1.08 


1.69 


1 . 66 


1 . 54 


Mind 


4.99* 


4.45* 


2.37* 


- 


— 










Carnival 


2. 10* 


1.92* 


1.46* 


.48* 


. 34* 


. 23* 


1.21* 


1.11* 


. 88* 



a "Fluency" means reflect the number of elements combined to generate a "product." 
^Differences significant at p < . 05. 
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Table 25 



Summary of Analyses of Main Effect of IQ: 
Posttest Variables 
(df for Effect = 2) 



Variable 



df for IQ effect .. . 

Error MS F 



Extended problems in convergent thinking 

Jewel problem: 

Achievement of solution 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 

House problem: 

Achievement of solution 

Number of ideas 

Number high quality ideas 

Quality rating, best idea 

Sensitivity to incongruous information 

X-ray problem: ^ 

Achievement of solution I 
Achievement of solution II 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 
Questioning and clarifying ability 

q 

Brief problem in convergent thinking 

Ingenuity test: 

Achievement of solution 

Extended problems in divergent thinking 3 

Ancient city problem 
Number of ideas 
Number high quality ideas 
Quality rating, best idea 
Questioning and clarifying ability 

A Visit to Karam 

Sensitivity to puzzling facts 
Intellectual persistence 



95 



216 



215 



204 



95 



98 



.57 


2.50 


20. 37 


3. 50* * 


3.59 


3.85* 


2.41 


3. 07 


3. 85 


22.72** 


20.67 


5.70** 


7. 14 


11.93** 


38.56 


24.79** 


5.91 


8. 45** 


. 05 


1.18 


1.24 


10.41 ** 


42. 18 


18. 07** 


9. 89 


1 3 . 45** 


12.44 


15. 04** 


4.75 


9. 10** 


100. 50 


3 5. 43** 


42.64 


7. 14** 


1.49 


4.68* 


3. 50 


5.26** 


5.21 


3. 44* 


.32 


<1 


13. 00 


6 . 09** 



a Problems in this group are all verbal. 

^These two variables represent a breakdown into separate solutions of the measure of achievement 
of solution reported for this problem in the earlier analysis. 

*p < . 05 

< . 01 



(Continued on next page) 
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Table 25 (c'ont’d) 
Variable 


df for 
Error 


MS 


IQ effect* 

F 


Brief problems in divergent thinking 

Circles (nonverbal): 

Number of ideas 


222 


198.30 


10. 07 * 


Number high quality ideas 




16.62 


5. 47** 


Quality rating, best idea 




.47 


2.49 


Dog improvement (verbal): 
Number of ideas 


222 


87. 04 


5. 01** 


Number high quality ideas 




4.96 


2.78 


Quality rating, best idea 




.75 


2.49 


Intelligence pill (verbal): 
Number of ideas 


223 


20. 90 


7.96** 


Inventiveness 




.23 


3.44* 


Summer carnival (verbal): 
Number high quality ideas 


223 


1.80 


4.23* 


Quality rating, best idea 




2. 11 


3. 85* 


Ability to synthesize 




8. 57 


6 . 56 ** 


Special tests 3 
Statue problem: 

Effectiveness of Master Thinking Skill 


110 


6.37 


2. 50 


Intellectual persistence 




1.11 


<1 


Insight and understanding of thinking: 


Sentence I 


74 


. 94 


9. 37** 


Sentence II 


85 


. 14 


2. 11 


Sentence III 


81 


. 17 


3. 01 



Problems in this group are all verbal. 

These two variables represent a breakdown into separate solutions of the measure of achievement 
of solution reported for this problem in the earlier analysis. 

*p < . 05 

**p < . 01 



Table 26 



Mean Level of Achievement of Solution for IQ Groups: 
Posttest Variables 



Problem 




IQ 




High 


Middle 


Low 


Jewel 


.67 


.49 


.34 


House 


. 86 * 


. 78* 


. 43 * 


X-ray a 


. 07, . 29* 


. 03, .12* 


. 03, . 04* 


Ingenuity 


4. 37 * 


3.59* 


2. 03* 



a These two variables represent a breakdown into separate solutions of the measure of 
achievement of solution reported for this problem in the earlier analysis. 

^Differences significant at p = .01. 
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Table 27 



Comparison of Ideas Generated by IQ Groups: 
Posttest Variables 



Problem 




Fluency 




No. 


hiah aualitv 


Quality of best 


H 


M 


L 


H 


M 


L 


H 


M 


L 


Jewel 


7 . 00* 


6.23* 


5.49* 


1.52* 


1 . 46 * 


.89* 


2. 52 


2. 28 


1.89 


House 


4. 23* 


4.71* 


3.68* 


1.33* 


1.38* 


.83* 


4.68* 


4. 49* 


3. 35* 


X-ray 


3.41* 


3.13* 


2. 00* 


1.19* 


. 99* 


.48* 


3. 11* 


2.65* 


2. 24* 


Ancient city 


5. 88* 


5. 03* 


3. 86* 


.61* 


.41* 


. 17* 


1.3 0* 


1.10* 


.69* 


Circles 


15. 13* 


14. 36* 


12. 06* 


2.37* 


2.18* 


1.49* 


1.82 


1.78 


1.68 


Dog 


8.65* 


7.68* 


6 . 5 5 * 


1.29 


1.24 


.83 


1.62 


1.58 


1.44 


No. of ideas 


3.24* 


2.79* 


2.22* 


— 


— 


— 


*■“ 






Pill; 

Inventiveness 


.11* 


. 04* 


.01* 


- 


- 


— 


— 


**■ 




Carnival 3 


2.° 10* 


1.92* 


1 . 46 * 


.62* 


. 47 * 


.32* 


1.30* 


1. 24* 


1— • 
• 

o 

o 

-X 



a,, Fluency" means reflect the number of elements combined to generate a "product." 
^Differences significant at p < . 05. 



Table 28 

Means for Remaining Posttest Variables for which Significant IQ Effect was Found 



Variable 



House problem: Sensitivity to 
incongruous information 

X-ray problem: Questioning and 
clarifying ability 

Ancient city problem: Questioning 
and clarifying ability 

A Visit to Karam 

Intellectual persistence 

Insight and understanding 
Sentence I 



IQ 

H M L 

.92 .55 .38 

.87 .51 .39 

1.30 1.10 .57 

3.16 2.84 1.77 

'.76 .29 



94 



Table 29 



Means for Significant Interactions Involving IQ and Treatment 



Variable 




H 


IQ 

M 


L 


Jewel problem: 










No. high quality ideas 


Treatment 


.78 


. 32 


• 2 1 




Control 


.38 


. 32 


. 26 


Squares problem: 










Number of ideas 


Treatment 


11.40 


7.20 


6. 57 




Control 


10.51 


9.40 


5.20 


Pill problem: 










Inventiveness 


Treatment 


.23 


. 05 


. 03 




Control 


. 00 


. 03 


. 00 


Summer carnival problem: 










No. high quality ideas 


Treatment 


.70 


. 38 


. 23 




Control 


.25 


. 30 


• 24 





Table 30 








Means for Significant Interactions Involving IQ and Environment 










IQ 




Variable 




H 


M 


L 


Truck problem: 
Number of ideas 


Facil. Env. 
Nonfac. Env. 


8. 05 
10. 39 


8.60 
8. 85 


6.47 

5.60 


Summer carnival problem: 
Quality rating, best idea 


Facil. Env. 
Nonfac. Env. 


1. 22 
1. 38 


1. 02 
1.47 


1.10 

0.89 


X-ray problem: 
Number of ideas 


Facil. Env. 
Nonfac. Env. 


3. 03 
3.79 


2.82 

3.44 


2. 18 
1. 81 
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IV 

DISCUSSION 



EFFECTS OF TREATMENT 

Mean performance of the group which re- 
ceived Series One of The Productive Thinking 
Program surpassed that of the Control group on 
30 of the 40 internal and posttest measures. 

Of the 13 significant differences, 11 favored 
the Treatment group. Clearly, then, use of 
the program led to an increase in the level of 
productive thinking in this large and representa- 
tive group of fifth-grade students. Moreover, 
this superiority was found at all three levels 
of IQ. It was evident on a broad variety of 
the measures, including tasks which were quite 
different from the problems used in the lessons 
themselves. The practical and theoretical im- 
plications of these findings merit further dis- 
cussion here. 

First, while the performance of the Treat- 
ment group generally exceeded that of the 
Controls, it did so only by a modest margin— 
rarely did mean scores for the Treatment group 
exceed those of the Control group by a factor 
of more than 20%. Differences of this magni- 
tude, although they may be both statistically 
and educationally significant, are considerably 
smaller than the dramatic differences obtained 
in two previous stu s (Covington and Crutch- 
field, 1965) using a., earlier version of Series 
One. In those studies the Treatment group 
sometimes scored more than twice as high as 
the Control group on certain measures. The 
present study, however, differs from those 
previous ones in several important respects, 
and failure to find such large differences here 
is readily understandable. One important dif- 
ference concerned the amount and ki,"d «f 
teacher participation during training. In the 
earlier studies, the classroom teachers were 
deliberately instructed to supplement the les- 
sons of Series One by featuring them in class 
discussions, by assisting and encouraging 
their students to transfer the thinking princi- 
ples to other curriculum areas, and by making 
the lessons a regular component of the overall 
educational effort. Recent evidence (Blount, 



Klausmeier, Johnson, Fredrick, and Ramsay, 

1967) indicates that even a moderate degree of 
teacher participation (e. g. , writing informative 
comments on student papers, providing encour- 
agement, etc. ) can increase the effectiveness 
of programed material by as much as 50%. This 
facilitative effect would probably be heightened 
even more by the greater extent of teacher par- 
ticipation which occurred in the earlier experi- 
ments. 

In contrast, the present study was de- 
signed to probe the instructional limits of the 
materials by using them as an entirely self- 
contained program, with all forms of teacher 
participation purposely held to a minimum. 
Moreover, the materials were administered at 
the rate of nearly one lesson per day— the fast- 
est pace feasible for regular school use. This 
fast-paced teacherless condition represents a 
very severe test of any instructional program; 
indeed, it is difficult to imagine a program 
normally being used under such restrictive con- 
ditions as these. By way of comparison, the 
most recent experimental study with Series One 
featured individualized supplementary activities 
and distributed practice, in which the Program was 
used over a much longer period of time. Results of 
this study are similar to those of the earlier studies, 
with performance of the Treatment group surpass- 
ing that of the Control group by a substantial 
margin. 

The results of the present -experiment, then, 
in conjunction with those from the other studies 
just mentioned, indicate that some degree of 
teacher participation (even if only modest) re- 
sults in much greater educational benefit from 
the lessons, even though the materials alone 
do produce significant differences in thinking 
and problem-solving performance. 

A second noteworthy aspect of this treatment 
effect is its generality. Superior performance 
of the treatment group was evident on a broad 
variety of measures and was not restricted to 
any one aspect of performance. A diverse set 
of performance indicators, each reflecting a 
different aspect of the total problem-solving 
process, showed consistent benefits as a 
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result of training. These included achievement 
of solution to problems, number and quality of 
ideas produced, intellectual persistence, sen- 
sitivity to discrepant or puzzling facts, use of 
a Master Thinking Skill, and an understanding 
of the process of thinking itself. 

However, the pattern of results did reveal 
some limits to the generality of training. These 
limits appear to be a function of the kinds of 
problems, rather than the kinds of performance 
indicators, employed. Specifically, there was 
a lack of treatment effects on nearly all per- 
formance measures for the brief problems, most 
of which emphasized divergent thinking. These 
brief tests, such as those developed by Torrance, 
are structured so as to elicit a large number of 
rather short, discrete responses on relatively 
uncomplicated tasks. Under these conditions, 
it seems likely that a student's level of per- 
formance would be affected considerably by 
his disposition or "set" to respond in a worth- 
while manner. Presumably, a positive set was 
present for the instructed students in this study 
as a result of training, but performance of the 
Control group may have been equally facilitated 
by the presence of a positive set which re- 
sulted from the novel and engaging quality of 
the tests and from the "escape" they provided 
from normal classroom routine. Thus, the ef- 
fect of set per se might not lead to differential 
performance in the two groups. 

Regardless of the effects gf set, however, 
one would still predict that the training of 
relevant cognitive skills would result in supe- 
rior performance by the Treatment group. One 
possible explanation for the lack of such re- 
sults focuses on the question of the relevance 
of such skills. If it is assumed that the brief 
tests require less complex cognitive processes 
for adequate performance than those processes 
taught for in the complex problem settings of 
Series One, then what is learned in Series One 
is only marginally relevant to performance on 
these brief problems. Thus the Treatment group 
might not show any marked superiority on such 
tasks. Another possibility which must be con- 
sidered is that most children of this age, 
whether trained or not, can demonstrate these 
relatively simple cognitive skills on demand 
(under proper conditions of short-term motiva- 
tion), even though they may rarely make use 
of them in their regular school work. The more 
complex tasks, on the other hand, presumably 
require the use of advanced skills which are 
not yet well developed, but which are learned 
and practiced in Series One. If this is true, 
training provided by the program would produce 
superior performance on complex thinking tasks, 



where the use of simple cognitive skills and 
a positive set is not sufficient for successful 
performance, but no obvious benefit from 
training would be apparent on the brief tests 
employed here. 

Clearly, there is a need for further research 
into (1) what cognitive skills are already well 
developed in children this age, skills which 
would require practice rather than special train- 
ing or instruction which was designed to create 
and establish them in the child's repertoire in 
the first place, and (2) what skills do not yet 
exist in the child's repertoire and will require 
specific and perhaps extensive training before 
the student will be able to make use of them. 
Further efforts devoted to the development of 
criterion measures should incorporate such 
research and reflect this distinction, if it is 
found to be valid. 

Yet another important aspect of the treat- 
ment differences is the occurrence of a gradual 
increase in the magnitude of these differences 
as training proceeds. The means presented in 
Table 15 show a very small difference in favor 
of the Treatment group on the first internal test 
(the Rare Coins problem), which was adminis- 
tered just before the half-way point in the 
training program. On the second internal test 
(the Deep Sea Dive problem), administered 
near the end of training, performance of the 
Treatment group exceeded that of the Controls 
by a greater margin, even though it did not 
reach statistical significance. However, on 
the posttest measures, administered when all 
training had been completed, performance of 
the Treatment group surpassed that of the Con- 
trol group by a statistically significant margin 
on two of the three relevant measures (i. e. , 
measures comparable to those used in the 
internal tests). There was, then, a steady 
increase in the effectiveness of training as 
the student proceeded through the lessons. 

This result differs from previous findings 
(Covington and Crutchfield, 1965), where the 
treatment effects were nearly as great on the 
first internal test as they were on the posttest. 
However, the conditions of teacher participa- 
tion in that study (discussed above) may have 
accelerated the effects of training. Thus, an 
important effect of teacher participation may 
be to increase the rate as well as the amount 
by which cognitive skills are developed when 
materials such as these are used. 

EFFECTS OF ENVIRONMENT 

The five significant Environment effects on 
the pretest measures all resulted from the fact 
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that classes with Facilitative environments 
performed better than Nonfacilitative classes 
on the two brief divergent thinking problems 
developed by Torrance. The probable sensi- 
tivity of these problems to the effects of set 
has already been mentioned. If this interpre- 
tation is extended to the present data, the 
significant Environment effects may possibly 
have resulted from the positive or negative set 
induced by the two types of classroom environ- 
ment. Indeed, in this study a Facilitative en- 
vironment was defined as one in which an 
overall positive set seemed to be present while 
the class was working on a divergtnt thinking 
task. Hence these pretest findings support 
the validity of the Environment classifications. 

On the complex problem-solving tasks, in 
contrast, an increase in the student's readi- 
ness" to think should not be sufficient to raise 
the level of performance, and in fact these 
problems did not show a significant Environ- 
ment effect. 

On the posttest and internal test measures, 
there was only one significant Environment 
effect out of 40 comparisons. This virtual 
absence of posttraining Environment effects 
seems to occur in large part because the treat- 
ment was particularly effective with students 
in Nonfacilitative environments, differentially 
raising the performance of this group and thus 
reducing any initial differences favoring the 
group having a Facilitative environment. 

Nearly a dozen Treatment X Environment inter 
actions (five of which are significant and are 
presented in detail in Table 19) support this 
conclusion. It should be noted that the spe- 
cial benefits of training for students in Non- 
facilitative classes amounted to more than 
simply a change of "set, " since some of the 
significant Treatment X Environment interactions 
involved the kind of complex problems in which 
no main effect of Environment was found. Thus 
it appears that an important effect of the train- 
ing materials was to improve the cognitive 
skills and general attitudes of students from 
the less favorable classroom environments, 
even in the absence of direct teacher interven- 
tion. Further research is needed to determine 
precisely what teachers of varying capabilities 
and attitudes can do, in conjunction with the 
training materials, to produce varying amounts 
of pupil gain or change. 

EFFECTS OF SEX 

Significant sex differences occurred on 21 
of the total of 61 pretest, posttest, and internal 
test measures used in this study. With the ex 



ception of three measures in the pretest Pit 
problem, all significant sex effects resulted 
from the superior performance of females. 

This female superiority occurred on all types 
of problems (both verbal and nonverbal, con- 
vergent and divergent), and on all performance 
indicators except measures of intellectual per- 
sistence and of the achievement of solution to 
problems. This finding of generally superior 
performance by females is consistent with 
other findings (e.g., Klausmeier and Wiersma, 
1964) concerning the verbal performance of 
students of this age. 

The absence of any significant Treatment X 
Sex interactions is particularly noteworthy, 
indicating as it does that Series One of The 
Productive Thinking Program was equally ef- 
fective with both boys and girls (although the 
girls managed to maintain their initial advan- 
tage throughout the training period). 

EFFECTS OF IQ 

The general level of intellectual ability had 
a clear and significant effect upon performance 
for most of the measures used in this study. In 
most cases, the predictable rank-order relation 
between IQ and performance was found, with 
the High IQ group obtaining scores superior to 
those obtained by the Middle IQ group, which 
were in turn superior to those obtained by the 
Low group. The effect of IQ was significant 
on all types of performance measures, and on 
all types of problems except the special test 
of the Master Thinking Skill. 

Large and significant IQ effects were found 
on all of the brief divergent thinking problems, 
and this effect was especially marked on the 
Squares problem, which is one of Torrance s 
nonverbal divergent thinking te Significant 
IQ effects were also found on the other three 
Torrance tests, and these effects were often 
large in an absolute sense (Tables 25 and 27) 
as well as significant statistically. This set 
of findings suggests that early reports of a 
"virtual lack of relationship between measures 
of creative thinking and IQ" (Torrance, 1962) 
are perhaps something of an overstatement and 
that such findings may have resulted from 
sampling restrictions. Since all IQ analyses 
in the present study are based upon the care- 
fully selected subsample of 240 students 
(described on page 3) which should be repre- 
sentative of the entire distribution of ability 
in a total of 44 classrooms, the highly sig- 
nificant IQ effects obtained cannot easily be 
dismissed as an artifact of the particular 
sample used. 
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Indeed, it seems reasonable to assume that 
there is a moderate correlation between IQ and 
creative thinking when the entire distribution 
of intellectual ability is considered. Most 
studies which have reported the absence of 
such a correlation (e.g. , Getzels and Jackson, 
1962; Wallach and Kogan, 19 65) involved highly 
selective, nonrepresentative samples, while 
studies which reported a positive relation be- 
tween IQ and creative thinking (e.g. , Ripple 
and May, 1962; Guilford and Hoepfner, 1966) 
have used far more representative samples. 

In any case, IQ had highly significant effects 
upon performance for the vast majority of 
measures used in the present study. 

In contrast to the effect of IQ on test per- 
formance, there was no consistent pattern of 
interactions between IQ and Treatment; that is, 
the increments in thinking and problem-solving 
performance produced by the training program 
were approximately equal for all three levels of 
IQ. This finding agrees with the results of the 



earlier Covington and Crutchfield study. It 
should be pointed out, however, that although 
the materials themselves may be used effectively 
with students at many different levels of intel- 
lectual ability, the kind of teacher participation 
required is very different for various levels of 
ability. Students who have serious reading dif- 
ficulties obviously require a great deal of as- 
sistance with the materials, while teachers 
whose classes are above average in general 
ability may devote most of their effort to show- 
ing the students how to extend and transfer the 
thinking principles to other curriculum topics. 

In an average class, the self-administering 
feature of the materials makes it possible for 
the teacher actively to assist a small group of 
students while the rest of the students work 
individually. Generally, though, the lower the 
level of student ability, the greater amount of 
teacher participation required for the effective 
use of these materials. 
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Uncle John notices Jim's silence: 



Jim, 



the reason you can't think 
of other ideas is that you first 
narrowed down to just one 
possibility. . . 



l^rTlyou see, you jumped to the 
conclusion that someone stole 
the water. You believed so. 
strongly in your idea that it 
blinded you to other 
ideas . 



I guess I really stumbled 
when I jumped, this time. 
Gee, it's hard to learn not 
to jump to conclusions'. 




Copyright © 1966 by Martin V. Covington, Richard S. Crutchfield, and Lillian B, Davies. 
py y w Reprinted with permis-sion. 
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Here’s another thinking guide that will 
give you a method for discovering many 
of the different ideas about this problem. 

\r 




/pick out each of the important things 
" in the story- -each object and person. 
Then take each of these things one at 
a time, and try to figure out how it 
might have had something to do with 
the disappearance of the water. 



This method will make sure that 
you don’t miss any important part 
of the problem that could give 
you ideas. 




Now, what will happen as Jim and Lila take Uncle John’s advice? Turn the 
page to find out. 
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You try making a list, too. Go back to pages 8 and 9 and read the story 
again. Then pick out each of the main things in the story and write it 
down: 
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